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/\BSTRACT 

N ,«vv» 

A discrete  time  transform  kes-beeivstudied  and  applied  to 
the  representation  and  discrimination  of  digitized  signals.  The 
transform  consists  of  an  orthogonal  (Hodamord)  matrix  whose 
elements  ore  all  ones  and  minus  ones.  To  focilitote  implemen- 
tation,  a fost  Hadamard  tronsform  (FHT)  h««-beeiva*''eloped 
requiring  only  NIogN  rather  than  algebraic  additions.  ii 

Several  properties  of  the  FHT  are  reveoled,  including  the  nature 
of  its  presence  in  the  fast  Fourier  tronsform,  in  which  it  per- 
forms the  additive  operations  as  shown  by  further  ^composing 
the  product  of  motrices  representing  the  FFT,.gAmono  other 
properties  discussed  ore  the  Hodamard-sampleS^Walsh  relation- 
ship, the  Hadamard  analog  to  the  "shift"  property  of  the 
Fourier  transform,  and  the  structure  of  covariance  matrices 
diogonolized  by  the  Hadamard  transform.  The  efficiency  of  the 
Hadamard  transform  is  compared  with  thot  of  optimum  transforms 
for  signal  representation  ond  discrimination.  For  signal  repre- 
sentation, the  optimum  transform  is  the  set  of  eigenvectors 
which  diagonalize  the  sample  correlation  matrix  of  the  signals. 
For  signal  discrimination,  the  optimum  transform  for  Nilsson's 
likelihood  decision  technique  is  that  which  simultaneously 
diogonolizes  two  different  covariance  matrices.  The  inductive 
capability  (training  set  vs.  unknown  dota)  of  discriminonts  esti- 
mated from  the  tronsform  coefficients  by  parametric  techniques 
is  described.  Discriminants  based  on  sample  meons  generalize 
better  than  those  based  on  probability  densities. 


INTRODUCTION 

The  impact  of  the  fast  Fourier  transform  on  signal  process- 
ing is  just  beginning  to  be  felt,  particularly  the  "fost"  convolu- 
tion property  of  the  transform.  In  this  paper,  a discrete  time 
transform,  termed  the  Hadamard  transform,  is  analyzed  and  some 
relationships  between  it  and  the  discrete  time  Fourier  transform 
are  developed.  In  addition,  the  effectiveness  of  the  tronsform 
for  two  applicotion  areas  of  interest,  signal  representation  and 
discrimination,  are  considered  in  detail.  Potential  application 
areas  best  suited  for  the  transform  are  suggested  and  contrasted 
with  those  being  exploited  by  the  fast  Fourier  transform. 

DISCRETE  TIME  FOURIER  TRANSFORM 

The  primary  transform  in  signal  analysis  has  been  histori- 
cally the  Fourier  transform  pair, 

F(w)  = f f(t)e-j*»dt 

*'•  nc 

and  its  inverse. 


For  signals  thot  ore  of  on  echo  or  burst  nature,  extending 
over  some  finite  time  epoch  0 to  T,  it  is  customory  to  consider 
the  signols  as  periodically  extended  functions  of  period  T for 
which  the  above  reduces  to  the  Fourier  series. 


J 


with  inverse 


T^ 


f(t)  e'j"*0'  dt 


WQ  ' 
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T 


f(t)  * 


We  know,  in  addition,  from  the  sampling  theorem^  that  a 
signal  bond  limited  to  W cycles  over  the  interval  0 to  T con  be 
represented  by  a 2TW  dimensionol  signal  vector's  where 
■5  = (s^,  S2,  S3,  ...,  S2Tyy).  The  increased  use  of  sompled 
and  digitized  signals  for  spectrum  analysis  hos  led  to  the  fur- 
ther modification  known  as  the  discrete  time  Fourier  transform^ 
defined  as: 

N-1 

, - V'  „-j2itnk/N  N = 2TW 

'n  ■ Z-t  n = 0,1 N-1 

k=0 


with  inverse. 
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The  Fourier  transform  becomes  an  N by  N unitory  matrix  F 
performing  as  a linear  operator  on  the  sampled  time  series  to 
obtain  the  "spectrum”  vector  f, 

T = Ft, 

where 

F F*  = F*  F = Nl. 

The  scope  of  this  poper  will  include  the  relationships 
between  the  Fourier  matrix  F and  on  N by  N orthogonol  matrix, 
H,  known  os  a Hadamard  matrix  to  be  described  next. 


DISCRETE  TIME  HADAMARD  TRANSFORM 

The  Hadamard  matrix^  is  generated  recursively  from 
successive  Kronecker  (direct)  products,  H(,j  = H2  X H|q/2  for 


H4  = H2: 


1 1 1' 
-1  1 -1 
1 -1  -1 
-I  -I  M 


It  is  clear  that  both  the  orthogonal  and  symmetric  properties 
o(^  H2  ore  preserved  in  eoch  step  of  the  recursion  so  that 

is  bo :h orthonormal  and  symmetric  = I and 


H = H'. 

In  the  analysis  of  the  rest  of  this  paper,  we  will  be  con- 
cerned only  with  N * 2^,  P = 1,2,3,...  os  the  dimensionality 
of  both  the  F and  H matrices. 

A matrix  T which  relates  F and  H can  be  computed  from  the 
product  F H since  from  F = TH,  FH"^  = T = The 

structure  of  the  T matrix  is  rather  obscure,  however,  and  a much 
simpler  and  appealing  relationship  will  be  shown  to  exist 
between  the  fast  Fourier  transform  and  a “fast"  Hodamard 
transform  to  be  developed  next. 

FAST  HADAMARD  TRANSFORM  (FHT) 

The  recent  innovation  of  the  fast  Fourier  transform  in 
digital  signal  processing  has  reduced  the  requirement  for  com- 
puting the  Fourier  transform  by  orders  of  mognitude  from 
real  products  and  adds  to  2NlogN.  A similar  algorithm  exists 
for  the  Hodamard  transform,  the  nature  of  which  we  discuss 


To  develop  the  algorithm,  we  note^, 

Diag  (Hn,  Hn)  = <2  ^ 

and  note  that  H2  X l|q  = > 

l|q  — N by  N diagonal  matrix.  Then  a recursion  formulo  can  be 
developed  from  the  following  property  of  direct  products: 

Hn  = H2XHN/2  = (I2XHn/2)(H2XIn/2) 


H4  = (I2XH2HH2XI2)  = (0^  y (1^  .1^) 


Hg  » (I2XH4HH2XI4) 


H2XI2  0 WI4  I4 
0 H2XI2M14  -I4 


continuing  we  have 

Hi6  = (I2  X Hg)  (H2  X Ig) 

= (Ig  X H2)  (I4  X H2  X I2)  (I2  X H2  X I4)  (H2  X Ig) 
ond  in  general 

logN  logN 

Hn  = n ('2l09N-i  X H2  X I i.i)  = TT  E i . 
i = l,2,3,  i = 1,2,... 

Since  is  symmetric  as  well  as  each  matrix  E^j  in  its 
product  decomposition,  it  follows  thot 

logN 

Hn  = TT  Eji 

i = 1,2,3, 

= (Hn)T 

= ...  E4T  E2T 

= En  ...  E4  E2  . 

Note  that  there  are  N add  opeartions  in  each  E matrix  with 
logN  such  matrices  in  each  product  giving  NIogN  algebroic 
operotions. 

We  next  show  the  marked  similarity  of  the  above  recursion 
to  one  which  generotes  the  fost  Fourier  transform. 

FAST  FOURIER  TRANSFORM 

To  do  this  we  return  to  the  matrix  F in  more  detail,  devoting 
our  attention  primarily  to  the  complex  form  of  the  Fourier  trans- 
form. Hamming®  and  athers  have  analyzed  the  finite  Fourier 
transform  as  a real  orthogonal  matrix  whose  rows  consist  of  the 
sampled  sines  and  casines,  but  the  Fourier-Hadamard  relation- 
ship is  more  cumbersome  to  develop  in  this  form. 

If  we  let  W * then  W"  becomes  the  n*^  harmonic 

(n*^  row)  of  the  discrete  Fourier  transform;® 


N-1 

k=0 


n = 0,  1 N-1 


* (l4XH2)d2XH2Xl2)(H2Xl4) 


2 


and  for  N = 4,  F = 


w»  wO 

W°  W'  W2  yy3 
yyO  w2  wO  ^2 

VwO  w3  w2  w’i 


where 


where  W"'‘  = 


Note  that  F Is  symmetric  since  reversing  the  exponents  of 
W"*'  in  the  summation  does  not  alter  the  values  of  the  terms  in 
F.  To  illustrate  the  FFT  using  direct  products,  we  alter  for 
convenience  the  designotion  of  W to  lower  cose  w = e'J  '"so 
that  w^'^"  becomes  the  n*^  root  of  unity. 

>=2  = (*o  J/2)  = (J  .1)  = H2 

by  rearranging  the  row  ordering  for  F^  it  con  be  shown  that 
(designating  Ffj  = P F|s|  where  Pisa  symmetric  row  permuta- 
tion matrix  to  be  discussed  later). 


where 

and 


F'  = h ''"I 

4 \g2  G2I 

G2  “ ^2  ^2  ' ^2 


/wO  0 \ 

U w’'4/ 


using  the  symmetry  properties  of  the  rotating  vector 


- 


where 


.-j2«(0)/4 

/ 

g-j2n(l)/4 

• (o^ylrD^jCi:  .4)  ■ 02»h,)6,(Hjxi,) 

D2  — Diag  (12/  ^2) 


in  general  one  can  show  the  recursion  for  the  Fourier  tronsform* 

Pn  ' ('2  ^ '^N/2)  '^N/2  (^^2  ^ 'n/2> 

* Since  completion  of  this  paper,  it  has  been  found  that  a 
similar  recursion  is  also  developed  by  M.  C.  Pease  in  the 
Jour.  A.C.M.  April  1968,  pp.  252-264. 


In  X (a,A2A3)  = On  x a,)  (i^  x A2)  (i  ^x  A3) 

one  has  for  example: 

Fg  = l2x[d2XH2)D2(H2Xl2)]D4(H2Xl4) 

= (I4  X H2)  (I2  X 62)  (I2XH2XI2)  64  (H2X  I4) 
where  I2  X D2  = Diag  |l,l,w®,w'/^, 1,1, w®,w^'^^j 

64  • Diag  [1,1, 1,1, w0,w’/®,w2/8,*3/8j_ 

The  matrices  present  in  the  FHT  can  be  seen  to  be  embedded 
in  the  product  form  of  the  FFT  above  for  N = 8.  Eoch  doubling 
of  N contributes  two  new  terms  and  modifies  those  already 
present  by  the  direct  product  with  I2.  The  general  product  term 
becomes: 

' = PFn 


N 


logN 


= P 


P n ('2logN-.  XD2M)('2<o9N-,XH2Xl2;.,) 

i = l,2,3 

logN 

TT  ^2'  ^2' 

i = 1,2,3 


where  E : make  up  the  FHT.  This  expansian  is  equivalent  .0 

2 n 

the  "decimation  in  frequency”  form  of  the  FFT."  Since  Fj^  is 
a symmetric  matrix  one  can  again  reverse  the  ordering  (noting 
also  that  E^j  = E^j"^  and  U^j  = U^i"^  since  U^j  is  diogonal). 


(logN  \T 


jT  . 


= En  Un  ...  E4  U4  E2  U2  P. 

This  form  of  the  FFT  has  been  programmed  in  FORTRAN  II 
with  provision  made  for  optional  computotion  of  the  FHT. 

Each  matrix  in  the  product  form  of  the  FFT  derived  by 
Brigham  and  Morrow^  and  by  Anderson^  can  also  be  split  into 
two  factors  where  each  product  pair  is  E^j  P D^iggiq.i  P. 


Before  proceeding,  consider  the  permutation  matrix  P which 
related  F to  F after  eoch  iteration.  We  con  designate  the 
harmonic  number  of  each  tow  of  F^  by  subscripting  that  tow 
vector  with  the  equivalent  binary  number  thus  the  2’'*^  harmonic 
of  F4,  the  row  vector  (w®  w^^^  w®  is  designated  fiQ 

/ w®  w®  \ 

\ 0 1/2/ • Each  recursion  in 

the  FFT  "doubles”  the  harmonic  number  to  obtain  even  har- 
monics in  the  top  half  of  F|q  and  adds  one  to  eoch  to  obtain  the 
respective  odd  harmonics  in  the  lower  half;  thus  F^  and  Fg 
obtained  by  the  recursion  has  the  following  ordering: 


/fo\ 

Then  F 2 consists  of  / = 


>=8 


^000 

1 

U.  1 

hoo 

fpo  = [0 

^010 

•ho'V 

—[110 

|pi  = 

|poi 

hi  = h 

hoi 

ipii 

hii 

In  each  case,  it  is  evident  that  the  natural  ordering  is  the 
reverse  ordering  of  each  binary  subscript  in  the  above  orroys 
(see  also  references  6 and  9).  The  permutation  matrix  P which 
reorders  the  Fourier  coefficients  to  natural  order  has  in  each 
row  0 1 in  the  column  defined  by  the  above  relationship.  Thus 
for  N = 4 


00/1  ooo\ 
01/  00  1 0 I 
' io\  01 0 oi- 
iiyoooi/ 


A further  discussion  of  the  P matrix  with  its  significance  in 
the  Hadamard-sompled  Walsh  relationship  is  presented  in 
appendix  A. 


COMPARISON  WITH  OPTIMUM  TRANSFORMS 


SIGNAL  REPRESENTATION  AND  DISCRIMINATION 

The  Fourier  tronsform  provides  o useful  and  familiar  refer- 
ence with  which  to  compare  and  relate  the  properties  of  the 
Hodomard  transform.  There  ore  several  application  oreas, 
however,  such  os  signal  representation  and  discrimination  for 
which  neither  transform  is  "optimum"  in  the  sense  of  efficiency 
of  representation  or  classification  accuracy. 

For  this  reason,  the  Hadamord  transform  was  compared  also 
with  the  following  two  transforms:  the  orthogonol  matrix  con- 
sisting of  the  eigenvectors  of  the  somple  correlation  matrix  of  0 
set  of  digitized  echoes,  and  a nonsingular  motrix  which  performs 
the  simultaneous  diogonolization  of  two  different  covariance 
matrices.  The  first  transfarm  is  ogtimum  in  the  mean  square 
sense  for  signal  representation  ' while  the  second  is 
optimum  within  the  context  of  0 specified  decision  criterion 


known  as  Nilsson's  likelihood  scheme  to  be  described  later  in 
this  paper.  The  aptimality  property  follows  directly  from  the 
attainment  of  stotisticolly  independent  votiobles  through  the 
diogonolization  procedure. 

A.  Signal  Representation: 

1.  Representation  of  signal  sets 

It  is  known  that  the  most  efficient  set  of  basis  functions 
for  representing  a doss  of  signols  is  determined  by  the 
Korhunen-Loeve  expansion. For  sampled  and  digitized  sig- 
nals, X « (x^  ...  x|q)  where  N = 2TW,  the  optimum  coordinont 
system  is  reodily  shown  to  be  the  eigenvectors  of  the  sample 
correlation  matrix  B, 

B = ((bjj))  i « 1,2,...,N  andj  » 1,2 N 

where  the  correlation  matrix  is  estimated  from  an  ensemble  of 
n column  signal  vectors 'x|,  using  the  relations 

n 

B = |-^*k  *k^  “"<<  b|j  = E }x|  xjj  . 
k»l 


For  0 choice  of  dimensionality,  r < 2TW,  the  coefficients 
of  the  signols  expanded  over  the  eigenvectors  corresponding 
to  the  r largest  eigenvalues  of  B will  represent  the  signols 
with  leost  loss  of  energy  on  the  overage.  To  show  this 
variational  property  one  finds  the  stationary  v'-lues  of  the 
quadratic  form 


Q(x)  = xT  B X , 

where 'x  - column  vector  subject  to  the  constraint 
0 >=  N(x)  - x^^x  - 1 , 
by  extremizing  the  functional 

H(l0  . Q(I^-AN(x)  . 

From  calculus  of  variations,  the  set  of  Euler-Lagronge 
equations  ore:  ^ 

° " 

from  which  ^ 

^ (X^jf)  * 2x;,  Ond  Cx^Bx)  * 2 ^ ' b|;  X| 
dx:  ' AX:  'J  ' 

N 

*j  ' ° ‘ ^ 

J*’ 

or  Af  • W. 
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The  actual  maximum  is  of  course  since  for  any  eigen- 
vector, If  follows  from  this  that  the 

squares  of  the  projections  of  the  signal  vectors  on  the  eigen- 
vector, y,  corresponding  to  are  maximized  due  to  the 
nature  of  the  correlation  matrix.  That  is,  let  the  signals  be 
column  vectors ‘x'|,...,'x„  then, 

n n 

»mox  = y^By  = 

k=1  k=l 

the  expression  on  the  right  being  the  sum  of  the  squares  of  the 
signal  vectors  projected  on  7. 

Tor  a comparotive  measure  of  the  representation  efficiency 
obtained  with  the  Hadomard  transform,  the  ratio  "rg"  between 
the  overage  energy  retained  by  a subset  of  Hodamord  coeffi- 
cients and  the  total  energy  (full  set  of  coefficients)  was  com- 
puted for  several  signal  sets  with  dimensionality  2TW  = 32. 

The  some  information  for  the  eigenvectors  of  the  correlation 
matrix  exists  in  the  corresponding  eigenvalue  spectrum. 

A comparison  of  the  number  of  coefficients  required  by  each 
of  the  two  tronsforms  to  retain  the  same  amount  of  signal  energy 
on  the  average  (r^)  is  a measure  of  the  relotive  efficiency  of  the 
Hadomard  transform. 

Spectral  plots  of  the  eigenvectors  and  the  Hadomard  codes 
are  shown  in  figure  1 for  o representotive  data  set  consisting  of 
52  signals.  The  relative  amplitudes  for  the  Hadomard  codes  ore 
the  mean  of  the  squares  of  the  values  of  the  11  largest  coeffi- 
cients. For  the  eigenvectors  the  function  illustrated  in  this 
figure  is  the  relative  mognitude  of  each  eigenvalue  plotted 
ogainst  its  position  in  a list  of  the  11  largest  eigenvalues  when 
ploced  in  order  of  decreasing  values. 

The  efficiency  is  such  that  82^  of  the  sum  of  the  expected 
squares  of  the  coefficients  of  oil  52  signol  vectors  making  up 
the  data  set  ore  contained  in  o three-dimensional  space  repre- 
sented by  the  projections  of  the  signals  across  three  eigenvec- 
tors. In  comporison,  the  binory  coordinate  systems  required 
seven  components  to  achieve  equol  representation  for  this  data 
set. 

Figure  2 shows  graphically  the  reconstruction  efficiency 
for  one  particular  echo  in  the  signal  set. 

The  reconstruction  of  o signal  based  on  the  most  represen- 
tative coefficients  of  a set  of  signals  tended  to  overemphasize 
the  lower  frequency  Hodamord  rows  possibly  due  to  a "washout' 
effect  of  the  signol  ensemble  on  the  relatively  large  number  of 
rows  with  considerable  high  frequency  spectral  content.  One 
finds  from  the  Hodomord-Wolsh  relationship  (oppendix  A)  that 
the  spectre  of  each  of  fully  half  of  the  functions  in  the  Hodamord 
transform  is  in  the  form  of  the  lower  sidebands  of  one  of  the 
other  half  modulating  the  sampled  voices  of  ’•'100...0  (equivalent 
to  a sampled  cos  Nnt  carrier,  0<t <1). 


The  eigenvector  corresponding  to  A^^j,  tends  to  resemble 
(but  not  equal)  the  mean  of  a group  of  positive-going  signals, 
thereby  retaining  a modest  amount  of  high  frequency  informotion 
present  in  the  mean. 

In  conclusion,  the  representation  efficiency  of  the  Hodamord 
tronsform  (based  on  mean  square  criteria)  was  found  to  be  sur- 
prisingly good  relative  to  the  eigenvectors  of  the  sample 
correlation  matrix. 

2.  Representation  Efficiency  of  o Single  Signol 

Several  interesting  observations  can  be  made  regarding 
the  Fourier  vs.  Hadomard  representotion  of  a single  signal. 

By  thinking  in  general  terms  again  about  the  Fourier 
spectra  of  sampled  Walsh  functions,  one  envisions  each  of 
those  which  modulote  the  sampled  '•'100...0  "carrier”  as  being 
linear  combinations  of  the  high  frequency  Fourier  components 
of  the  signal  with  bandwidth  2W.  Therefore,  while  a signal 
having  considerable  energy  spread  over  the  higher  frequency 
range  is  by  definition  not  efficiently  represented  by  its  Fourier 
coefficients,  it  often  is  by  its  Hadomard  coefficients. 

In  the  extreme,  signals  which  undergo  spectral  widening 
transformations,  such  os  clipping  or  phase  modulation,  ore 
often  more  efficiently  represented  by  their  Hadomard  coeffi- 
cients. 

In  figure  3-A,  a signal  with  comparative  Fourier  and  Hoda- 
mord  reconstruction  efficiency  (using  eight  coefficients)  is 
shown,  with  the  Fourier  line  spectra  of  one  of  the  best  Hada- 
mard  coefficients  shown  in  figure  3-B.  In  figure  4 the  recon- 
struction of  a signal  with  2TW  = 256,  using  the  FHT,  is  shown, 
using  the  best  32  coefficients. 

SIGNAL  DISCRIMINATION,  THE  PATTERN  RECOGNITION 
PROBLEM 

The  problem  of  discriminating  between  two  or  more  classes 
of  signals  can  be  subdivided  into  two  distinct  phases,  the 
estimatian  problem  and  the  classification  problem.  In  the  esti- 
mation phase,  the  parametric  technique  for  arriving  at  a dis- 
criminant configurotion  was  chosen. 

The  porametric  opprooch  consists  of  the  techniques  of 
statistical  estimation,  estimation  of  probability  densities, 
computation  of  sample  means,  etc.  The  statisticol  parameters 
are  computed  using  o set  of  data  known  os  the  troining  set,  and 
the  results  are  considered  as  estimates  of  the  true  parameters 
describing  a larger  class  of  dato  generated  by  mechanisms 
assumed  to  be  similar  to  those  which  generated  the  training 
data.  This  larger  class  represents  the  real  world  or  working 
phase  in  the  pattern  recognition  problem. 

The  degree  of  sophistication  employed  (i.e.,  sample  means 
vs.  probability  densities)  is  usually  determined  by  how 
accurately  these  parameters,  as  estimated  from  the  training  set, 
are  actually  expected  to  describe  the  unknown  doto.  This 


generalization  problem  is  of  central  importance  in  any  pattern 
recognition  problem. 

Three  estimation  and  discrimination  schemes  were  studied 
in  regard  to  their  capabilities  with  the  training  set  only,  and 
with  a small  degree  of  generalization  (50%  training  set,  50% 
unknown),  using  approximately  250  signals,  each  having  a 
dimensionality  of  32  time  samples. 

These  methods  were: 

a.  The  matched  filter,  based  on  sample  means. 

b.  Discriminants,  derived  from  independent  probability 
density  estimates  (simultaneous  diagonolization  of  two 
different  covorionce  matrices). 

c.  Discriminants  derived  from  probability  densities  at  the 
outputs  of  the  Hadamord  transform. 

ESTIMATION  BASED  ON  SAMPLE  MEANS 

Matched  Filter:  The  matched  filter  (linear  discriminant) 
configuration  consists  of  the  difference  of  the  mean  vectors  of 
the  two  classes.  The  term  ‘‘matched  filter, “ is  a specialization 
in  the  sense  that  only  the  correlation  for  zero  delay  or  dot 
product  between  the  signal  vector  ? and  a stored  vector  is 
implied  rather  than  o full  convolution  operation. 

The  estimation  problem  involves  merely  the  computation  of 
the  sample  mean  vector  for  each  class  in  the  training  set.  The 
classification  problem  for  which  this  procedure  is  optimum  is 
that  in  which  the  covorionce  matrices,  K;,  are  alike  and  are 
assumed  to  be  scalar  matrices.  The  i'"  class  consists  of  a 
mean  vector  m|  in  a 2TW  dimensional  signal  spoce  surrounded 
by  0 spherical  cluster  of  vectorsTfj,  j*l,2,...,r  each  representing 
mj  perturbed  by  independent  additive  Gaussian  noise  with  zero 
mean  (figure  5-A).  For  this  poper,  we  will  consider  only  dis- 
crimination between  two  classes,  i = ),2.  Since  the  covorionce 
matrix  for  the  multivariate  Gaussian  distribution  is  the  some  for 
each  class,  the  conditional  o priori  probability  density  is 
T W 

j(x-mj)''’ K-’  (x'-Sj)! 


to  the  scalar  terms  which  are  functions  of  the  energy  content  of 
the  mean  vector  mj.  Often  amplitude  invariance  is  desired  so  o 
modified  (not  necessarily  optimum)  discriminant  is  employed, 
resulting  from  the  normalizotion  of  mj  (division  by  ||mj||  i-1,2). 
The  resulting  correlation  discriminont  x"^-  (mi  - m2)  <0  yields 
decisions  affected  only  by  signal  shape  information  (direction 
and  not  length  of  x os  in  figure  5-C). 

If  y = Hx,  and  U|  - H nij,  o discriminant  with  similar  proper- 
ties operating  on  the  Hadamord  mean  vectors  ui  and  u2  can  be 
expressed  by  y"^  • (ui  -02)  <0,  where  ||ffj|l  -\/N  Hitijll.  In 
addition,  since  the  Hadamord  (or  any  nonsingulor)  transform  on 
the  data  effects  a similarity  transform  on  the  covariance  matrix, 
K,  it  is  left  unchonged, 

/LI-1\T  /)  Li-1  1 I LI  uT  ) I 


(H-’)''’(Jjl)H-l 


IihhT 


The  actual  discriminant  can  often  be  simplified  by  truncat- 
ing the  vector  representing  the  difference  of  the  normalized 
mean  vectors  u]  and  U2  of  the  two  dosses  making  up  the 
training  set.  In  figure  6-A  a discriminant  is  shown  (in  the  time 
domain)  resulting  from  the  linear  combination  of  those  Hoda- 
mord  rows  corresponding  to,  and  weighted  by,  the  seven  largest 
elements  of  d,  where  d -T)i  — U2-  A comparison  of  this  vector 
with  the  vector  d,  figure  6-B,  shows  the  small  effect  of  trunco- 
tion  for  this  data  set  (data  set  1).  Note  the  relative  lack  of 
high-frequency  detail  in  the  discriminant,  a property  which  will 
be  shown  later  to  be  in  marked  contrast  to  the  spectral  content 
of  discriminants  based  on  probability  density  estimates 
(simultaneous  diagonolization). 

ESTIMATION  AND  DISCRIMINATION  USING  PROBABILITY 
DENSITIES 

The  more  general  problem  of  discriminotion  between  two 
multivariate  Gaussian  processes^^  with  estimated  parameters 
N(mj,Kj),  i = 1,2,  involves  the  determination  of  theconditional 
probabilities  p(ltji) 

_ ] I Lro*  _ Si  \T  i/.-l  /.e  _ - 


p(x|i)  - exp  -- -(x-fh-)'  K ''  (x  - rfi;) 

(2n)''’|K|r2  12  'I 

with  Ki  ^ K2  in  general.  The  log  likelihood  decision  for  equal 
p(i)  becomes 


i2Tw-'J®">ify g(x)  - in-^4r  < ° O'  * Q2  - Qll  < 0 

p(xl2)  2[  iKil  ^ ' 


j2  = NqW  Nq  ^ noise  power  spectral  density.  ^ obo^^. 


The  log  likelihood  decision  is  g("x)  = In  p ® 

The  surface  g(lf)  » 0,  is  a hyperplane  separating  the 
clusters  in  figure  5-A  between  the  means  till  and  m2.  The 
likelihood  function  is  simply, let  <T  * 1,  for  convenience 

gCx)  « . (mi  - m2)  - mi  • mi  + m2  • ^2]  <®' 

An  inspection  of  figure  5-B  and  the  above  expression  for  two 
classes  shows  that  the  decisions  ore  amplitude  sensitive  due 


A linear  nonsingular  transformation  A on  the  dato  vector  x 
results  in  the  vector  y with  Gaussian  parameters  N(u|,Z|)  where 
Z|  results  from  a similarity  transformation  on  Kj  by  A.  Of 
particular  theoretical  interest  is  the  transformation  R which 
simultaneously  diagonalizes  Ki  ond  K2  resulting  in:^^ 

g(?)  * *^2^  ^ ^2  ~ • n]  < ® 

where  A is  K2  diogonolizcd,  Tj=R(x-mj)  and  Ki  is  tronsformed 
to  the  identity  matrix.  The  coefficients  of  this  transform  can 
then  be  processed  os  independent  univariate  Goussian  variables. 
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An  application  of  this  procedure  in  an  as/mptotic  sense  to 
the  class  of  covorionce  matrices  describing  noncyclic  stationary 
processes  was  described  by  Capon^^  who  employed  sampled 
power  spectre  obtained  from  the  Fourier  transform. 

One  observes  the  presence  of  o bios  term  ond  energy  meas- 
urement terms  in  the  likelihood  function  above,  resulting  in 
amplitude  sensitive  decisions,  an  undesirable  feature  in  many 
pattern  recognition  applications.  Furthermore,  one  of  the 
exploitoble  feotures  of  the  Hadamord  transform,  from  an  imple- 
mentation stondpoint,  is  its  binary  nature.  Because  of  this  it  is 
often  desiroble  to  process  the  Hadamord  coefficients  as  random 
variables,  by  clipping  them,  followed  by  an  estimation  scheme 
requiring  only  the  area  under  the  positive  portion  of  the  proba- 
bility density  function  of  each  coefficient. 

Such  a scheme  has  been  reported  by  Nilsson^^  which  in  turn 
develops  a set  of  weights  by  which  the  clipped  coefficients  of 
unknown  signals  are  linearly  combined  as  a discriminator.  To 
develop  the  weighted  discriminant,  the  assumption  of  stotistical 
independence  between  the  variables  is  made  which,  for  the 
usual  case  of  nondiagonal  covaiiance  matrices,  is  unjustified. 
Because  of  the  implementation  simplicity  of  the  method, 
however,  it  was  desirable  to  compare  the  results  of  using 
Nilsson’s  approoch  on  the  coefficients  of  both  the  Hadamord  and 
"R”  transform,  the  latter  satisfying  the  requirement  of  statisti- 
cal independence. 

Nilsson’s  method  is  summarized  here. 


SIMULTANEOUS  DIAGONALIZATION 

While  the  "R”  matrix  is  known  to  be  the  Fourier  transform 
for  stationary  processes,  most  data  encountered  in  pattern 
recognition  problems  belong  to  the  more  general  doss  of  non- 
stationary processes.  For  such  processes  a matrix  is  derived^^ 
which  performs  a lineor  transformation  of  the  signal  data  in  such 
a way  that  the  covariance  matrix  of  doss  1 is  reduced  to  the 
identity  matrix  while  the  covorionce  matrix  describing  class  2 
is  diagonalized.  For  class  1,  the  random  variables  obtained  by 
correlating  each  row  of  the  filter  matrix  with  the  signals  will  be 
independent  with  unit  variance  about  the  transformed  meon.  For 
class  2 the  outputs  ore  again  statistically  independent,  but 
with  variances  proportionol  to  the  respective  eigenvalues.  The 
technique  is  accomplished  in  the  following  manner. 

The  parameters  of  o set  of  multivariate  Gaussian  si  gnals,  X, 
are  described  by  a mean  vector  ifi]  and  a covariance  matrix,  R, 
where 

R . i -1,2 N and  j .1,2 N . 

The  covariance  matrix  can  be  estimated  from  a set  of  sig- 
nals using  the  relation: 

‘'ij  - ^ l^t^i " ""i'  ("j  - < 

In  the  procedure,  one  computes  the  product 

R2  R]'  where  the  motrices  R]  and  R2  arise  in  the 
following  manner. 


Let  p(l)  . p(2),  and  consider  from  Bayes  theorem  the  ratio. 


pdliO 
P (2|S) 


P (x|1)  < 1 
P(S|2)  > 


or  g(x) 


InPML)  < 0 

P ati2)  > 


where  x — transform  coefficient  vector.  With  statistical 
independence  we  hove: 


P (x|1)  - I I P (Xjll)  etc. 

i-1 


By  clipping  the  transformotion  coefficients,  the  notation 
complexity  reduces  from  the  analog  case  to  the  following: 

pfxj.llD  ^ Pi  , p(x|  .011)  ^ 1-Pi 

p(xi-ll2)  ^ g;,  p(xi.0l2)  ^ l-g; 


which  is  a linear  discriminant  estimated  from  the  cumulative 
probability  for  X|. 

Note  that  the  presence  of  the  bias  term  does  not  introduce 
amplitude  sensitivity  in  the  decision,  due  to  clipping.  For  0 pre- 
determined transform,  (Hadamord,  Fourier)  the  clipping  also 
renders  the  estimation  scheme  amplitude  insensitive  on  a signol- 
to-signal  basis. 


We  let  R2  be  the  covariance  matrix  of  class  2 and  then 
determine  R]-1/2  by  first  diagonalizing  the  covariance  matrix 
(R])  of  class  1.  The  i*^  column  of  the  column  eigenvector 
matrix  Q,  which  performs  the  diagonalization,  is  multiplied  by 

-i  (^1  — eigenvalue  of  R^)  yielding  the  matrix  R]'1/2^ 

•JXj 

known  os  the  whitening  filter  for  doss  1,  since 
(R,-’/2)T  r,  r,-1/2  , I. 

The  symmetric  matrix  (R'l'^'^^)'^  R2  results  when  the 

transformation  which  whitens  the  covariance  statistic  of 
class  1 is  applied  to  the  covariance  matrix  of  class  2.  When  0 
second  transformation  (M)  is  found  that  diagonalizes 
(R]'1^^)1^  R2  it  can  be  seen  that  it  will  not  affect  the 

statistics  of  class  1,  since  M^^IM  = I where  M^M  = I,  the 
matrix  M consisting  of  the  column  eigenvectors  of 
(R]'1^^)^  R2  We  then  obtain,  finally,  0 linear  trans- 
formation R which  is  equal  to:  ^ 

X = QHMy  where  QH  = r'^ 
or  y = (QHM)-’  X 

y = R X 
R « mI"  H-’  qT, 

The  R matrix  then  yields  a set  of  32  time-domain  filters 
which  yield  outputs  that  are  statistically  independent  for  both 
classes  and,  thus,  could  be  thought  of  as  the  optimum  set  of 
lineor  observables  for  discriminatino  between  two  stationary  or 
nonstationary  Gaussian  processes.  " 
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Figure  7 shows  one  row  of  the  "R”  tronsformotion  obtained 
from  the  above  procedure  for  a typical  pair  of  classes.  Note 
the  predominant  high  frequency  content  in  the  spectra  of  the 
filter  which  is  in  direct  contrast  to  the  motched  filter  discrimi* 
nants. 


Table  1 shows  a typical  set  of  relative  discrimination 
accurocies  for  studies  involving  220  signals  (2TW  s 32),  with 
g^  = 0.5.  The  eight  best  coefficients  estimated  from  the 
training  set  for  each  tronsform  were  used.  As  before,  while 
absolute  accuracies  are  not  indicated,  all  exceeded  50%. 


This  phenomenon  was  noted  in  all  rows  of  the  “R”  trans* 
formation  motrix  and  occurred  consistently  for  all  data  sets 
studied.  It  appears  to  be  reloted  to  the  decrease  in  stotistical 
dependence  with  increosing  frequency  between  the  coefficient^^ 
of  the  Fourier  expansion  of  a stationary  Gaussian  time  series. 

A comparison  of  the  effectiveness  of  the  Hadamard  ond 
“R"  transforms  tor  signal  discrimination  is  illustroted  in 
figure  8 for  a representotive  signol  set  (2TW  = 32),  consisting  of 
35  signals  from  class  1 and  35  from  class  2 (data  set  1).  The 
Nilsson  scheme  of  estimation  ond  classification  wos  employed 
on  the  tronsform  coefficients  m both  cases.  The  relotive 
classification  accuracy  is  plotted  agoinst  the  number  of 
coefficients  with  largest  weights  in  Nilsson’s  method  used  to 
form  the  discriminont.  The  lorgest  subset  of  coefficients 
shown  is  eight  out  of  o possible  32.  Although  the  absolute 
scale  is  not  shown,  oil  occuracies  exceeded  50%. 

While  the  difference  between  the  best  coefficient  of  each 
transform  is  not  representative  of  all  data  sets  studied  the  rate 
of  increose  and  resulting  divergence  of  the  accuracies,  os 
extra  coefficients  ore  odded  to  the  discriminant,  is  typical  of 
the  dato  and  is  due  to  statistical  dependence  between  the 
Hadamard  coefficients. 

Although  studies,  performed  with  the  training  set  only, 
provide  useful  clues  on  the  relative  effectiveness  of  probability 
density  measures  on  the  various  transform  coefficients,  the 
generalization  properties  of  the  estimation  procedures  is  of 
considerable  importance  ond  is  described  next. 

GENERALIZATION  PROPERTIES 

Defining  o generalization  ratio  g^  by: 

No.  of  signals  in  training  set 

No.  of  unknown  signals  (working  phase) 

the  Hadomord  transform,  “R”  tronsform,  and  matched  filter 
discriminant  were  evaluated  ogainst  the  troining  set  only 

(g^  = 1*0)  ond  for  g^  = 0.5. 

For  the  “R”  tronsform,  a technique  sufficient  for  the  pur- 
poses of  testing  relative  generalization  capability,  wos 
employed  by  summing  selected  *'R"  transform  coefficients. 

This  scheme  (suggested  by  the  statistical  independence  of  the 
coefficients)  allows  for  prediction  of  the  classification  per- 
formance on  the  training  set  since,  for  n independent  Gaussian 
n n 

voriobles.  Vary  = ^ Vorj  ond  Uy  - Uj.  Clossification 
i=l  i=l 

occuracies  predicted  from  the  standord  deviation  tables  for  uy 
ond  (Vor)y  agreed  closely  with  the  accurocies  actually 
meosured. 


TABLE  1 

Comparison  of  Discriminont  Techniques 


Simultoneous  Diagonalization 

Matched  Filter 

Hadamard  (Nilsson  Technique) 

From  the  table  it  can  be  seen  that  significant  improvement 
over  the  matched  filter  in  clossification  of  the  training  set 
appears  possible  by  obtaining  independent  probability  meosures 
of  the  two  dosses.  The  generalizotion  copobility  of  the 
matched  filter,  however,  exceeded  that  obtained  by  the  “R** 
tronsformotion  for  the  data  studied,  possibly  due  to  the  differ- 
ences in  spectral  content  of  the  two  discriminants.  Although 
the  clipping  operation  in  Nilsson's  method  prevents  o time 
domain  reolization  as  o linear  discriminant,  those  Hadamard 
codes  included  in  the  Nilsson  discriminant  exhibit  a more 
balanced  spectra  (high  and  low  frequency  content). 

The  FHT  was  applied  to  a set  of  3500  digitized  signals 
each  with  256  time  samples  under  more  severe  generalization 
conditions  (10%  training  set,  90%  unknown).  The  Nilsson  and 
the  matched  filter  discriminants  were  estimated  from  the  full 
tronsform  ond  employed  as  classifiers  on  the  best  32  Hadamard 
coefficients.  It  was  found  that  the  sample-means  discriminant 
was  agoin  more  effective  from  a generalizotion  standpoint  then 
those  estimated  from  probability  densities. 

CONCLUSIONS  AND  APPLICATION  AREAS 


Training  Set  Unknown 


L% 

25 

0 

12 

8 

11 
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The  Hadamard  transform  appears  to  have  useful  applicotion 
in  signal  representation  especially  In  the  areas  of  dato  com- 
pression systems.  Several  investigotors  have  noted  the  appli- 
cability of  the  transform  for  further  processing  the  power 
spectral  coefficients  of  the  channel  vocoder^^'^^  before 
transmission  over  o channel.  Such  studies  have  emphosized 
reduced  bit  rate  due  to  coorse  quontizotion  of  selected  coeffi- 
cients rather  than  truncation  of  the  number  of  coefficients. 

Severol  investigators^^  have  studied  the  relative  merits  of 
the  ‘tronsformotion  compression"  opprooch  with  other  methods 
of  date  compression,  studying  in  particular  the  Fourier  ond 
Korhunen-Loeve  techniques  using  the  coefficient  truncotion 
technique.  In  reference  23  it  is  concluded  that  the  two  tech- 
niques are  efficient  in  performonce  but  improcticol  in  imple- 
mentation. For  those  systems  considered  in  reference  23  which 
include  digital  sensors  or  A D conversion  of  the  doto,  it  would 
seem  thot  perhaps  the  FHT  would  alleviote  the  implementation 
problem. 
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In  signal  discrimination  it  would  appear  that  the  use  of 
probability  densities  on  selected  transform  coefficients  should 
be  employed  with  generolizotion  in  mind  for  signals  with 
appreciable  variability  in  shape  within  a class.  The  meons  of 
the  transform  coefficients  appear  to  be  useful  for  many  practical 
problems.  Often  in  pattern  recognition,  one  searches  for  trans- 
form properties  which  render  it  less  sensitive  to  a particular 
type  of  variation  of  a signal  such  os  translation,  change  in 
amplitude,  etc.  Such  attacks  against  the  generalization  problem 
tend  to  overshodow  in  importance  the  excellent  methods  and 
insights  given  to  os  by  theory  for  discriminoting  between 
training  sets  or  even  generolizing  on  the  basis  of  partially 
known  parameters. 


One  of  the  principle  applications  of  the  FFT  in  signal 
processing  is  in  the  performance  of  rapid  convolution  using 
frequency  domain  techniques'^  including  matched  filter  process- 
ing of  pulse  compression  radar  echoes,  etc.  Because  the 
Hadomard  transform  does  not  possess  these  translation-related 
properties  it  would  not  be  expected  to  play  a major  role  in  this 
area  (see  appendix  B). 

Orthogonal  multiplex  data  transmission  systems  have  been 
discussed  in  references  24  and  25,  with  some  comments  made  in 
the  latter  concerning  the  burst  error  immunity  properties  of  such 
systems.  It  would  appeor  that  the  FHT  would  have  application 
in  the  formation  of  the  transmission  signol  ond  in  the  receiver. 


APPENDIX  A 

THE  HADAMARD-WALSH  RELATION  AND  THE  SIGNIFICANCE  OF  P 


The  permutation  matrix  P can  be  shown  to  be  identical  to 
that  which  carries  the  ordering  of  the  rows  of  the  Hadomard 
matrix  to  the  matrix  whose  rows  are  the  sampled  Walsh  functions, 
termed  the  Walsh  matrix^^.  The  Walsh  functions  in  turn  have 
been  defined^^  over  the  interval  (0,1)  as  the  successive  modula- 
tion in  various  combinations  of  a set  of  square  waves  'f'p(t), 
p = 2",  n = 0,  1,  2,...r,  where  p designates  the  number  of 
periods  in  the  interval.  For  p = 0,  +0  (t)  ^ 1,  0<t<1. 

Replocing  p by  its  equivalent  r bit  binory  number,  the  waveform 
of  a particular  Walsh  function  is  specified  as  the  praduct  of 
those  Walsh  functions  represented  by  each  "1”  in  its  binary 
subscript.  Letting  r = 3 for  example, 'I' gn  = ('•'OOl)  ('•' Olo) 
seen  to  be  the  modulation  of  4'gQ'|  by  '•'fllQ  9). 

The  row  vectors  of  the  Walsh  matrix  of  order  N =2*^  ore 
designated  os+j,  b|  = 0,1,  and  can  be  generated  from  Q 

recursion  relation  involving  "ig  and  ♦] . For  N = 2, 

H2=  *here?Q  = (1,1),  = (l.'l)-  Then  it  follows 

that: 

hb,...ko  ■ V.1.0 ■ V -s**' 

fob,... bo  . ■ \...bo>'»0 

where  a X E = (aib-|,aib2,a2bi,a2b2)  "a  = (0^,02) 

b = (b],b2)- 


From  this  it  follows  directly  that 

n,..bibo  = ^ = 0,1 

for  example  '•'ng  t®sul*s  '*']g  by  the  operation 

♦no  = 'fg  X X *1  =(1  1)X(1-1)X(1-1)  = (1-1-1  11-1-1  1). 
At  the  same  time  it  can  also  be  shown  that 

V-blbg  = X ^b,  X +bg  . 

where  h|,  ...b^bg  *be  b„...b]bg  row  of  H. 

For  example,  h^^g  = X X 'I'g  is  the  only  nonzero 
row  of  E1H2  X E^H2  X EgH2  = (El  X E]  X Eg)  Hg 

where  El  =(°f),Eg 

Thus  the  same  permutation  matrix  specifies  the  Hadamord- 
sampled  Walsh  reordering  and  the  required  reordering  necessary 
in  the  FFT.  Because  of  this  equivalence,  other  interpretations 
exist  for  the  generation  of  each  Fourier  harmonic,  (row  of  F). 
For  example  if  N = 4 it  is  cleor  that  fg  is  the  complex  modula- 
tion of  f2  with  fi, 

Tgi  = Ti  = (w0,wi,w2,w3)  = (w0,w2)  X (W°,wl) 
flO  = T2  = (W0,w2,w0,w2)  = (w0,w0)  X 

w1w2  w2w0,w3w2^  = (w0,w3  w2,w'). 

APPENDIX  B 


PROPERTIES  OF  THE  HADAMARD  TRANSFORM  AND  FOURIER-HADAMARD  ANALOGIES 


There  are  three  properties  of  the  Fourier  transform  which 
have  contributed  greatly  to  its  usefulness  in  signal  processing 
applicotions,  the  "shift"  property  and  the  resulting  canvolution 
theorem37,28  being  perhaps  the  most  useful.  The  others  include 
the  invariance  of  the  power  spectrum  to  translotion  and  finally 
the  simultaneous  diagonalization  of  all  covariance  motrices 
describing  stationary  processes. 


The  "shift”  property  of  the  Fourier  trr  nsform  is  stated 
mathematically  be  the  statement, 

if  F(s)  =/“  f(t)e-j»»dt 
then  the  Fourier  transform  of  f(t  - t)  is  e'J''*  F(s). 

In  discrete  time  on  equivalent  express^n  for  the  shift 
property  is  the  following,  if  T=  F ■$,  then  f]  • F tW 
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and  W = ^ as  before 

Thenti  = Diag  [w^.w' W^-lj  T = D t 

The  “shift"  property  becomes  equivalent  to  the  specification 
F t'  = D F or  F T’  F*  = D 

thus  the  shift  property  is  exhibited  by  the  set  of  eigenvectors  of 
the  translation  operator  T.  By  expanding  |Al  - T|  = 0 one 
obtains  W',  i = 0,1,  ...,N  - 1 os  the  set  of  eigenvolues,  thus 
each  row  of  the  Fourier  motrix  is  an  eigenvector  since,  for 
example, 


Since  any  translations  less  than  the  period  is  expressible 
os  a power  of  T,  the  Fourier  matrix  is  the  set  of  eigenvectors 
diagonalizing  T and  thus  exhibits  the  shift  property  in  discrete 
time. 

The  invjrionce  of  the  power  spectrum  to  translation  follows 
directly  from  thms  property.  The  class  of  covariance  matrices 
diagonalized  by  the  Fourier  matrix  is  also  easily  specified  as 
linear  combinotinns  of  the  powers  of  T, 


N-1 

K = oqI  +2]  0|(Ti  +TN-i) 
i=  1 

which  is  identified  as  the  set  of  cyclic  stationory  processes.^® 


it  is  evident  that  the  Hodamard  transform  does  not  possess 
these  properties  with  respect  to  the  tronslotion  operation.  It 
does  possess  these  properties  with  respect  to  an  operator 
defined  by  the  ts/llowing  relation 


\ = (l  o)  ^ '2I09  N-k  = ’-2 ^ 

where  aH  designotes  the  Kronecker  power  of  A,  A X A ...  X A, 

k ^erms 

or  products  of  the  above  terms.  In  porticular,  the  linear  combi- 
nation of  such  products  defines  a class  of  covoriance  motrices 
diagonalized  by  the  Hadamord  matrix. 


LkJ 


Letting  I2 


this  follows  from 


= (H2[^  X H2N  N-k]  ) S|,(H2W  X H20°9  N-kj)_ 
H2MT2W  H2W  X (H2f'°9 


Using  the  property  p,  the  above  express 

becomes  (H2I2  H2) 

matrix  for  k 


^Oog  N - k] 

1,2,  ...,log  N.  For  example,  let  N 


X l2ll°9  N-KJ  which  is  a diagonal 


4,  then 


etc. 
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Figure  1.  Eigenvolue  • Hodomord  Spectra 


Figure  2.  Reconstructed  Efficiency 


AMPLITUDE 


Figure  7.  Simultaneous  Diagonal! zation  Discriminant 


Figure  9-B.  Walsh  Matrix  N « 8 


